This invention relates to a medical imaging real time viewing videoconferencing system, more particularly an apparatus and method of using said medical imaging videoconferencing system with multiple input operators or participant clients viewing each other's inputs collaboratively and concurrently.
Videoconferencing systems are becoming more commonly used to conduct meetings and share information, including in the medical field. Participants are typically geographically separated and wish to share ideas and thoughts as they participate in the conference. With such a videoconferencing system, audio and video signals are transmitted over a communication link, such as telephonic, to be reproduced at a remote videoconferencing system so the parties can see and hear each other. In many cases, the videoconferencing systems can support video images allowing each party to view moving camera images, as well as other screen displays. Videoconferencing systems are used in many different ways. Some of the most common is to share computer graphic presentations, such as a POWERPOINT® slide presentation where a user shares his or her slide presentation with others in the conference. The parties can also share video images. The operator uses the available conferencing system and Super Video Graphic Array (SVGA) as a method of viewing these video signals to document and provide camera images on the user's computer or laptop.
Past videoconferencing systems have many disadvantages, including but not limited to, if a participant has a question on a slide or aspect of the presentation, the presenter must control the images to scroll back to the location in question and must toggle through the slide presentation to answer the participants' question. Also, in most conferencing systems, the presenter has control over the presentation, and the participant has no control over what other participants can view. Past video conferencing that have any type of illustration feature requires the user to take a step to undo or to delete telestrations while in session; user must affirmatively act to delete input illustrations. However the present invention will auto erase the telestrations, drawings and annotations after a preset predetermined amount of time. They can also be temporarily turned off from auto fading, and the last auto faded telestration, annotation and drawing can be brought back to the same previous location, if desired.
Recently many inventors have seen the need to allow a plurality of clients or users to collectively collaborate on presented work. These systems allow two or more users of the internet to move or modify Hyper Text Markup Language (HTML) documents with referring to the same. These systems work with browsers and web sharing managers provided in the shared client computer system of a source and receiver, and are constructed in such a manner that the web sharing manager of the shared client computer system of the receiver can receive the event message of the source from the web sharing manager of that source. Accordingly the event message is shared by the source and receiver, and the displaying and controlling of the same web page are simultaneously realized on the shared client computer system. Even further still, as incorporated by reference U.S. Pat. No. 7,310,657 to Nakamura describing in summary, a computer system comprising a plurality of user systems connected to each other being adapted to display a work area on a display screen, alternatively a plurality of users' systems connected to each other though a computer network. In Nakamura user systems include: collaboration work controller having a user management table for registering a node identification code given for each of the user systems and owner identifier related to the node identification code, and an object management table for registering object information related to the node identification code; and an obtainer for obtaining, based on an event entry for an object, the node identification code related to the object by referring to the object management table, obtaining the owner identified related to the obtained node identification code by referring to the user management table, and displaying the object on the screen in the manner that the obtained owner identifier can be discriminated from owner identifiers of other objects. Nakamura shows a display screen where users are participating and collaborating in work drawing annotations simultaneously. The owner identifier identifies the user for each object the owner identifier is displayed to the user watching the display screen with the entry (drawing) of the object from the other user. In other words the owner can be identified; it is possible to identify the owner of the object of the collaborative work easily. Each client runs from their own system and does not work from a server but merely each computer runs individually over a network,
Also incorporated by reference herein in its entirety is U.S. Pat. No. 8,924,864 to Mariotti, which describes in summary, a Consultant Client or plurality of Consultant Clients having the capability to view still and streaming video medical images, a person user able to input illustrating over those images. Mariotti relates generally to a multimedia collaborative conferencing system and method of using the same for generating telestrations and annotations on medical images concurrently with other users. One feature Mariotti is that the client can draw, telestrate and annotate over the image as he/she sees fit. Mariotti U.S. Pat. No. 8,924,864 does not allow client drawing telestrations and annotations to self erase as the present invention does. Not having these illustrations self erase can obscure important surgical image detail. Also, self erasing is an advantage to a client who is performing a surgical procedure in a sterile environment at the same time as viewing the images and does not have immediate ability to affirmatively communicate with a keyboard, mouse, stylus, or touchscreen monitor to delete items.
However, in past systems the computer arrangement can be summarized as a plurality of users systems connected to each other, each being adapted to display a work area on a display screen or connected through a computer network. Collaboration of work is done on each system by use of a management table for registered node identification codes given for each system user. That is, every computer system, or one system, requires (as in Nakamura) storage of collaboration user identifier in at least one of the user's computer system. However the Mariotti reference U.S. Pat. No. 8,924,864 does not contain what the present invention contains which is the telestrations, annotations, drawing overlays do not disappear without the operator taking action, wherein the present invention will automatically fade and delete telestrations after a predetermined preset amount of time, for one or a plurality of Collaborator Clients. The present invention allows for telestrations, annotations and drawings to overlay onto images and videos, that disappear, fade and self-erase after a predetermined amount of time. These self-erasing telestrations are important when the TIMS system is used for medical intraoperative live real time surgery streaming of medical images. It allows a Consultant Client to telestrate and annotate and have another Consultant Client see the telestrations as the intraoperative surgery is happening. The telestrations disappear so details of the medical images or videos are not compromised to the viewer, providing a clear unobstructed view to maximize the image fidelity in order to make the best clinical decision for the patient. This allows for a Collaborator Client to view streaming video or capture images along with telestrations without having to clutter up the view, as the telestrations will disappear automatically. It is important when viewing live detailed streaming surgical procedures that the input illustrations do not obscure details of the surgical landscape. In addition, a Client can elect to temporarily turn off auto fading to keep a telestration, annotation or drawing visible when discussing a specific procedure within the surgical landscape and needs the telelstration, annotation, or drawing to remain over the landscape longer than the preset auto fading duration time. A Client can also bring back the last faded telestration, annotation, or drawing to the same previous location on the surgical landscape to continue their discussion or repeat the last instruction, if desired. All input illustrations are stored on the TIMS server and delivered to each Client's viewing window. The input illustrations auto erase when the preset duration time has expired. However, the last erased input illustration from each Client is stored on the TIMS server during the consulting session, so the Client can have the last faded telestration brought back to the surgical landscape and at the same location as it was previously, if desired, to continue the discussion or repeat the last instruction.
The Mariotti Reference concurrently collaborative communications for use with medical imagery by allowing a server to have master control allowing for faster and more efficient performance, as well as allowing for a collaboration with medical images. Concurrently collaborative communications device and method for use has improved upon the past art by allowing a server master control allowing for faster and more efficient performance, as well as allowing for a medical Digital Imaging and Communications in Medicine, hereinafter referred to as DICOM, environment. As this invention approves upon and applies in a concurrently collaborative environment and allows for each user to collaborate simultaneously with all users viewing every other users' work product, as the work product is being created, all coincident with image including video, and audio, wherein the medical image together with based on a server and the illustrations are appended to that image. The Mariotti Reference (U.S. Pat. No. 8,924,864) network system apparatus improves upon and applies in a collaborative environment which provides for capture, retrieval and concurrent viewing of both real time and archived medical imagery streams for communication, collaboration and consultation with one or more sources of streaming imagery data by one or more users, also known as participant clients.
The traditional way of capturing an image on a medical imaging apparatus commonly called a modality, generally consisted of an operator or technician first conducting a scan. Then, using the modality to save the image, in still or motion video format, into the modality memory or into a main image storage database. Soon afterward, perhaps downloading the image into a hospital database such-as a PACS system, Picture Archiving and Communications System, hereinafter referred to as (PACS) or PACS server, or medical imaging archives for storage and later retrieval. Hereinafter referred generically as PACS. PACS can be further defined by a storage and management system for medical images. Typically, pertaining to the medical field, images such as x-rays, MRI's and CAT scans require a greater amount of storage than other images in other industries. The doctor would then access the PACS system to retrieve the image, the doctor at that time would call up the image, view and review the image, and conceivably develop a diagnosis based on the information from the image. This system imagery is viewed by a user and diagnosis made without image delay and the user accomplishes all these tasks in real time. Real time referring to events simulated by a computer at the same speed that they would normally occur in real life. In graphics animation, for example, a real time program would display objects moving across the display at the same time they would actually move, or in the case of Mariotti, a collaborator client views the image in real time and collaborates from client to client with no perceivable delay to any client. The illustrated embodiment is comprised of three essential components (FIG. 1,2): one called Tele Medicine Imagine Management System (TIMS) Server, another called the TIMS Collaborator, and a third called the TIMS Streamer. The TIMS
Server is a computer that manages users, security, channels and sessions within the TIMS Collaborator™ System (i.e. this invention described herein) allows for multiple users in multiple locations to concurrently collaborate on the images, each user to input highlighted graphic electronic traces and annotations over the medical image is also a system that allows one or more users located remotely to the imaging modality, to analyze, discuss, and save such analysis or discuss in a clinically relevant manner. Each is able to view, and comment on each of the users input concurrently in real time. This self-erasing telestrations is important when the TIMS system is used for medical intraoperative live real time surgery streaming medical images, it allows the one Consultant Client to telestrate and annotate and have a another consultant client see the telestrations as intraoperative surgery is happening the telestrations disappears so details of medical image is not compromised to the viewer. Saving telestrations, drawings and annotations over the images is only accomplished if the telestrations drawing and annotations are visible and before any auto fading takes place. The TIMS Streamer server is a device that processes any video output from a video source into a stream. A stream is defined as at least one image frame that defines a time progression of output from a video source. The input illustrations can be saved if the pre-set preferences have been the self-erased telestration feature disabled or shut off. A Client can also bring back the last faded telestration, annotation, or drawing to the same previous location on the surgical landscape to continue their discussion or repeat the last instruction, if desired.
In Mariotti the TIMS Server provides the real-time video and audio communication, as well as a method of recording, transmitting and saving images in a single file format structure, including as specified in DICOM Standard. DICOM is a medical imaging standard common in the medical industry. DICOM can also be defined as a standard in the field of medical informatics for exchanging digital information between medical imaging equipment (such as radiological imaging) and ensuring interoperability with other systems. DICOM, including protocols for device communication over a network, syntax and semantics for commands and associated information that can be exchanged using protocols, a set of storage services and devices claiming conformation to the standard, as well as file format and medical directory structures to facilitate access to images and related information stored on media that shares information. The embodiment can serve as the connection point between any medical imaging modality and a hospital PACS, medical archive or other image repository. One component of this invention, the TIMS Server, is able to connect DICOM equipment and older non-DICOM equipment to a hospital network, allowing imaging studies to be stored and saved. The TIMS Collaborator™ System, this invention described herein, briefly described as a trace overlay and annotation system that users can collaborate with each other in real time, each viewing each other's object inputs and those object inputs can be encapsulated and saved in a single file format structure, including as specified in DICOM Standard, in PACS, in a DICOM compliant image archive, or in other image repositories. Saving images with input illustrations is done if the input illustrations are visible (and not disappeared), in other words the fade and diaper option is off or not selected. If auto erase is off the input illustration will remain visible until the user takes affirmative steps to erase or delete.
Mariotti has developed a novel and simple network system apparatus and method of using the same, to allow a group of persons to concurrently collaborate on a computer system, with each participant viewing each other's telestrations, drawings, and annotations and saving them together with streaming imagery data, and relevant imagery metadata, including appended imagery metadata and saving them together in a single file format structure as may be required by standards for clinical documentation or medical records storage, including as specified in the DICOM Standard. Now added the present invention is the ability to concurrently collaborate and view telestrations, drawings and annotations for only a preset predetermined amount of time, and bring back the last faded telestration, annotation, or drawing to the same previous location on the surgical landscape.
The invention relates generally to a multimedia collaborative conferencing system and method of using the same for generating input illustrations, which include telestrations, drawings and annotations on medical images concurrently with other users and having those input illustrations self-erase. However when input illustrations are visible, saving the participant client input illustrations with streaming imagery data, and relevant imagery metadata, including appended imagery metadata in a single file format structure, including as specified in the DICOM Standard can be accomplished. The network system apparatus in this invention, is the TIMS Collaborator™ System. It is comprised of three essential components, one called the TIMs Server, another called the Collaborator Client, and a third called the TIMS Streamer. The TIMS Streamer includes a medical image acquisition system adapted for receiving and transmitting medical images, constructed from, a computer having communications capability adapted for acquisition and transmission of a plurality of medical imaging and video signals. Wherein the medical image and video signals are acquired at the medical device's native resolutions, transmitting the signals at their native resolutions and native frame rates to a receiving device, receiving the medical imaging video signals in analog or digital form, and if required, compressing and scaling the signal, converting the signal to digital form for transmission, and transmitting the digital signals using secure encryption protocols to a display device. The TIMS Streamer is capable of concurrently acquiring signals from a plurality of medical imaging systems, as depicted in FIG. 1, including but not limited to, ultrasound, Computer Tomography (CT) scan, fluoroscopy, endoscopy, magnetic resonance imaging, nuclear medicine, echocardiogram ultrasound and microscopy. Medical imaging equipment is also referred to as modalities. A more complete list of sources for DICOM imagery streams can be found in the DICOM Standard [PS 3.3 Part 3: Information Object definitions.], which include video (imaging), audio (waveform), and clinical documents (structured reports).
The TIMS Streamer can also receive the video image signal from a plurality of video sources, including but not limited to, S-video, composite color and monochrome, component red blue green video (RGB, three additive primary colors), Digital Visual Interface (DVI), any video transport protocol including digital and analog protocols, high definition multimedia interface (HDMI, compact audio video interface uncompressed digital data), serial digital interface (SDI), and DICOM video in their native, enhanced or reduced resolutions or their native, enhanced or reduced frame rates. The component, known in this invention as the TIMS Server, manages the communication between all acquisition systems (TIMS Streamers), between all users (Collaborator Clients), between the hospital site server, located on site or remotely, that stores the hospital's images, and the hospital network in both local area and wide area configurations. The TIMS Server manages the real time streaming imagery data acquired from the TIMS Streamers, and archived imagery retrieved in a predetermined digital single file format structure, including as specified in DICOM Standard. A participant or user computer can be defined as typically made of several components such as a main circuit board assembly having a central processing unit, memory storage to store programs and files, other storage devices such as hard drives, and portable memory storage, a power supply, a sound and video circuit board assembly, a display, and an input device such as a keyboard, mouse, stylus pen and the like allowing control of the computer graphics user interface display, where any two or more of such components may be physically integrated or may be separate. In one depiction, a remote location communicates with the networked computer, for the purpose of collaborating and conferencing with medical streaming imagery data.
An apparatus and method for using the same for concurrent collaboration between users, collaborating by video, audio, telestrations and annotations, including collaborating on medical images that are typically accessed on a storage server database, imaging archives, or continuous streaming video. The streamer server is continuously streams images to the TIMS server. Any number of clients can request information from the TIMS server. Each client in a conference with another or other clients can view all the clients object inputs as they occur. A client includes a user, typically a person who has interest in using the system for medical review and diagnosis of patient image data. The TIMS server keeps track of all steaming devices that are available and show it as such to the consultant clients and collaborator clients, and any conduit between. The TIMS server provides continuity with the PACS system, and stores information on all clients The TIMS server includes: a manager, this component controls the streaming requests to the streamer; server and also manage authorization and authentication tasks for access and privileges; the TIMS server, an administrative interface that manages users information, roles, session information, TIMS server configuration, streamer server configuration, web services wherein the web service interacts with the consultant client and collaborator client, performing such services as—sending studies to PACS, retrieving studies from PACS, retrieving study and patient information from a DICOM modality worklist utility (DMWL), sending media for a collaboration session to the client; participants of that conference, PACS server configuration, text chat information; a DICOM send service, wherein the DICOM send service retrieves the list of studies from the PACS server and sends the studies to the clients. A study is defined as medical images and patient data combined. DICOM Modality Worklist is defined as a software utility that invokes DICOM query and retrieve functionality which enables imaging equipment (e.g. medical modalities) to query medical image repositories, including but not limited to PACS, and obtains details of patient and scheduled examinations electronically, including patient demographics and study data, avoiding the need to type patient information multiple times (FIG. 10).
The TIMS server also manages all the illustration overlays, specifically, the entire client sketches, drawings, telestrations and annotations. Illustrations, also known as collaborant annotations are defined herein as any user input such as but not limited to drawings, sketches, telestrations, voice annotations, letter character text and numeric character text (FIG. 7). All illustrations are managed by the TIMS server based on a file sharing scheme where new illustrations keep getting appended to the file on the TIMS server. If the input illustrations auto erase is turned off then all input illustrations can be saved at any time during the session. If Auto erase is turned on, the input illustrations will disappear after a preset predetermined amount of time, but can be saved before the end of the auto fade duration time. Only a copy of the populated file is locally maintained on each client's computer. This approach of appending files on the TIMS server significantly improves performance and reduces image wait time, unlike the prior art approach of each computer having to update the image file and send it. This process of multi-layer multi user illustration appending and updating on the TIMS server any underlying image, including video, without sacrificing bandwidth is novel to this invention. Appending the files to the server periodically can be performed in a synchronous manner or asynchronous manner. Moving images with illustrations back and forth from a computer to a server, results in losing illustration quality or consuming more bandwidth. However, this novel invention appends only the illustration file to the TIMS server. All client computers use a local registration method process, local to the frame used for viewing. Each client is able to use a scalable window so all illustrations are ratio metric based on the underlying image aspect ratio. Therefore, all the illustrations always point to the part of the window and image as originally intended, regardless of window size on the clients computer display. For example, from the TIMS server, the streaming images and the client's computers that are capturing those images are used in a collaborative session. First, the files are distributed to all the participating clients. A central frame counter originating in the client computer of the person who has play/pause control merely issues frame synchronization commands to synchronize the stream on all client's systems. This method significantly reduces bandwidth and improves responsiveness of the-system. Synchronization can further be optimized by periodic synchronization signals used by the session initiator/controller such that every so many frames are confirmed to be synchronized. The client computer also sends a synchronized command whenever the computer image is paused, thus ensuring that the same frame is available to all participating clients by broadcasting the pause frame number along with the pause command to all participating clients. Client participants can receive video streams directly from the streamer server using a local area network. The invention can also detect if a user has low bandwidth and can compensate by only sending selected frames of the image to that user. or if initiator pauses on a frame can send that frame to users. An example, with low bandwidth of this is that the server sends every third or fifth frame of a video to one client so that client does not have any perceivable delay due to low bandwidth. However, client participants using the internet must receive all streams from the TIMS server. This is done to add a level of security so images do not go over the internet without encryption.
Participants can take several roles. The users can view real time streaming; video, view selected real time streaming video capture real still or motion video; view in real time captured streaming video; add multiple channels dynamically (FIG. 4). In addition, Consultant Client application is a collaborative, interactive synchronous or asynchronous media annotation-system, which will be used in medical files, used to enable users to collaborate and interact on archived medical images for clinical review and discussions and deciding on relevant medical procedures. The Collaborator Client can perform all of the functions of the Consultant Client, as well as communicate with another or other participating clients, by adding annotations in text or drawing form, text chat, query, save, and retrieve studies to and from PACS, and retrieve study and patient information from a DICOM Modality WorkList server (DMWL).
Communication between components is summarized as follows: The client (either the Consultant Client or—Collaborator Client views a stream to the TIMS server (FIG. 9). The TIMS server sends that request to the streamer server. The streamer server provides the stream to the requesting client directly, if connected through a local area network or through the TIMS server, if connected to the internet. When the client sends a request to stop the stream, the TIMS server notifies the streamer server to stop streaming to that client. TIMS server communication with a PACS server is as follows: the client sends a request to the TIMS server to store and/or retrieve studies from—the PACS server. The DICOM send service either retrieves the file and sends it to the client, or sends a command from the client to the TIMS server and saves the study as a DICOM file and sends it to the PACS.
This invention allows for two different types of user participants as defined herein, first is a Consultant Client and the other is a Collaborator Client. The Consultant Client communicates with the TIMS server by retrieving live streams from the streamer server, and in some cases, can capture the stream and view it. Consultant Clients, which can view streams through setting up channels on the network, having the ability to freeze frame streams, illustrate and save. However, the Consultant Client does not collaborate over the network with others. The initiator of the session or host consultant client can illustrate over the image with other consultant clients merely viewing the image and illustrations. An example of this would be a teacher illustrating a surgical procedure to students who merely watch. The Collaborator Client communicates with the TIMS server by retrieving streams from the streaming server, saving and retrieving studies to and from a PACS server, and retrieving patient information from a (DMWL). Collaborator Clients can communicate with each other through collaboration sessions. The Collaborator Client has all the features of the Consultant Client, plus the added functionality of creating illustrations and annotations, while collaborating with others in a session, saving the collaboration session as a DICOM file (eg a dcm file) and have the ability to send the file to a PACS server or other destinations, saving input illustrations can only happen when input illustrations are not erased.
Users or participants, viewing images from a medical modality concurrently collaborate with each other through a collaborative session. These users or participants are also known as Collaborator Clients. Two or more users can concurrently collaborate in a given session. One user can initiate the collaboration session and the other users can enter the session and actively participate in the collaborative session. The initiator can share media (such as a medical modality image) in the collaboration session with it being visible to all participants. All users can add annotations on the media using the draw control on their computer. All the annotations added by any user are available and visible to all the collaboration participants. In addition, users can add telestrating drawings, text annotations, voice annotations, and video annotations to the collaboration as well. The preference properties settings in the software will allow user to set the auto fade duration time on the TIMS server. The auto fade option can be set to off when it is desired that all telestrations remain over the image until a Client sets the erase icon to erase one or several or all of the telestrations over the image. The auto fade option can be set to on allowing the auto fade to function. Also, within the preferences, the length of time the telestrations, annotations, and drawings appear over the image before fading can be set. Telestrating is defined herein as a device allowing its user to draw a line, circle, arrow, drawing or sketch over archived or live medical image, images and video. Furthermore, each participant can also use the system to chat with each other using a text chat facility. A separate text computer window box is displayed that allows for each user to instant message each other in text format in a separate window application. One feature of the present invention is that the initiator can disable the edit control of any participant, such that a particular participant will not be able to add or edit the annotation or telestration. At this point, that participant can only view the annotations made by others users. Another feature of the present invention allows the initiator to pass control of the video stream start/stop/pause functions to another participant. This control feature can be edited to enable or disable the functionality to all participants or selected participants and can be done at any time during the collaborative session.
The invention also works with personal digital assistants. Participants (PDA) clients can use these PDAs to view, consult and collaborate on DICOM images. Personal digital assistant is any small mobile hand held device that provides computing and information storage such as hand held computers, phones, media display devices with storage and palm top computers.
The principle preferred embodiment and modes of operation of the present invention have been described in the forgoing specification. The invention which is intended to be protected herein, however, is not to be construed as limited to the particular embodiments disclosed, since these embodiments are to be regarded as illustrative rather than restrictive. Variations and changes may be made by others without departing from the spirit of this invention. Accordingly, it is expressly intended that all such variation and changes which fall within the spirit and scope of the claims be embraced thereby.